Skip to content

[SPARK-24937][SQL] Datasource partition table should load empty static partitions#21883

Closed
wangyum wants to merge 5 commits intoapache:masterfrom
wangyum:SPARK-24937
Closed

[SPARK-24937][SQL] Datasource partition table should load empty static partitions#21883
wangyum wants to merge 5 commits intoapache:masterfrom
wangyum:SPARK-24937

Conversation

@wangyum
Copy link
Member

@wangyum wangyum commented Jul 26, 2018

What changes were proposed in this pull request?

How to reproduce:

spark-sql> CREATE TABLE tbl AS SELECT 1;
spark-sql> CREATE TABLE tbl1 (c1 BIGINT, day STRING, hour STRING)
         > USING parquet
         > PARTITIONED BY (day, hour);
spark-sql> INSERT INTO TABLE tbl1 PARTITION (day = '2018-07-25', hour='01') SELECT * FROM tbl where 1=0;
spark-sql> SHOW PARTITIONS tbl1;
spark-sql> CREATE TABLE tbl2 (c1 BIGINT)
         > PARTITIONED BY (day STRING, hour STRING);
spark-sql> INSERT INTO TABLE tbl2 PARTITION (day = '2018-07-25', hour='01') SELECT * FROM tbl where 1=0;
spark-sql> SHOW PARTITIONS tbl2;
day=2018-07-25/hour=01
spark-sql>
  1. Users will be confused about whether the partition data of tbl1 is generated.
  2. Inconsistent with Hive table behavior.

This pr fix this issues.

How was this patch tested?

unit tests

@SparkQA
Copy link

SparkQA commented Jul 26, 2018

Test build #93602 has finished for PR 21883 at commit 90f99fc.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jul 27, 2018

Test build #93633 has finished for PR 21883 at commit d684d38.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jul 27, 2018

Test build #93645 has finished for PR 21883 at commit b44e578.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum
Copy link
Member Author

wangyum commented Jul 27, 2018

retest this please

@SparkQA
Copy link

SparkQA commented Jul 27, 2018

Test build #93652 has finished for PR 21883 at commit b44e578.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum
Copy link
Member Author

wangyum commented Jul 28, 2018

cc @gatorsmile @gengliangwang

@wangyum wangyum changed the title [SPARK-24937][SQL] Datasource partition table should load empty partitions [SPARK-24937][SQL] Datasource partition table should load empty static partitions Jul 31, 2018
gatorsmile
gatorsmile reviewed Jul 31, 2018
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala Outdated
def getPathFragment(partitions: TablePartitionSpec): String = partitions.map {
case (k, v) =>
escapePathName(k) + "=" + escapePathName(v)
}.mkString("/")
Copy link
Member

@gatorsmile gatorsmile Jul 31, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TablePartitionSpec is a map. The output of this func might be different from the actual schema, right?

For example, the partition schema is [partCol1, partCol2]. Your output could place partCol2 before partCol1

Copy link
Member Author

@wangyum wangyum Jul 31, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right. Fixed it.

@SparkQA
Copy link

SparkQA commented Jul 31, 2018

Test build #93829 has finished for PR 21883 at commit 46bfcc6.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@gatorsmile
Copy link
Member

gatorsmile commented Jul 31, 2018

@wangyum How about our Hive tables? Add a test case for Hive tables? They are created by CREATE TABLE instead of CREATE TABLE USING. You can use csv/json format in the test cases.

@SparkQA
Copy link

SparkQA commented Aug 1, 2018

Test build #93855 has finished for PR 21883 at commit 536346e.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum
Copy link
Member Author

wangyum commented Aug 1, 2018

retest this please.

@SparkQA
Copy link

SparkQA commented Aug 1, 2018

Test build #93868 has finished for PR 21883 at commit 536346e.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@gengliangwang
Copy link
Member

gengliangwang commented Aug 1, 2018

retest this please.

@SparkQA
Copy link

SparkQA commented Aug 1, 2018

Test build #93886 has finished for PR 21883 at commit 536346e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@gatorsmile
Copy link
Member

gatorsmile commented Aug 1, 2018

LGTM

Thanks! Merged to master.

@asfgit asfgit closed this in 9f55860 Aug 1, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gatorsmile gatorsmile gatorsmile left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@wangyum @SparkQA @gatorsmile @gengliangwang